Multiplexing apparatus and demultiplexing apparatus

ABSTRACT

The multiplexing apparatus which does not make a user who watches a moving picture feel uncomfortable, the moving picture including: (i) a coding unit ( 42 ) for generating one or more coded streams so that no gap occurs at the connection of predetermined two access units in a process of decoding, in sequence, the two access units in all of the access units included in the coded streams; and (ii) a multiplexing unit ( 18 ) for multiplexing, with other information, the coded streams generated in the coding unit ( 42 ).

This application is a divisional of application Ser. No. 12/572,538,filed Oct. 2, 2009, which is a continuation of application Ser. No.12/406,361, filed Mar. 18, 2009, which is a divisional of applicationSer. No. 10/591,242, which is the National Stage of InternationalApplication No. PCT/JP2005/010453, filed Jun. 1, 2005, now U.S. Pat. No.7,526,031, issued Apr. 28, 2009.

TECHNICAL FIELD

The present invention relates to a multiplexing apparatus formultiplexing one or more coded streams including coded pictures withother information so as to generate multiplexed data and ademultiplexing apparatus for demultiplexing the multiplexed data. Thepresent invention especially relates to a multiplexing apparatus and ademultiplexing apparatus which can perform special playback such asmulti-angle playback or the like.

BACKGROUND ART

Recently, the multi-media era has come in which sound, pictures andother pixel values are integrated into one media, and conventionalinformation media as communication tools like newspapers, magazines, TV,radio and telephone are regarded as the targets of multi-media.Generally, multi-media is a form of simultaneous representation of notonly characters but also graphics, sound, and especially pictures. Inorder to handle the above-described conventional information media asmulti-media, it is a requisite to represent the information digitally.

However, it is unrealistic to directly process a huge amount ofinformation digitally using the above-described conventional informationmedia because, when calculating the data amount of each informationmedium described above as digital data amount, data amount per characteris 1 to 2 bytes while that of sound per second is not less than 64 Kbits(telephone speech quality) and that of moving pictures per second is notless than 100 Mbits (present TV receiving quality). For example, a TVtelephone has already become commercially practical thanks to IntegratedServices Digital Network (ISDN) with a transmission speed of 64 kbps to1.5 Mbps, but it is impossible to transmit moving pictures of TV cameraas they are using ISDN.

That is why information compression technique is necessary. For example,a moving picture compression technique standard of H.261 or H.263 whichis recommended by the International TelecommunicationUnion-Telecommunication Standardization Sector (ITU-T) is used for TVtelephones. Also, with the information compression technique of theMPEG-1 standard, it becomes possible to store image information,together with sound information, in a normal CD (Compact disc) formusic.

Here, Moving Picture Experts Group (MPEG) is an international standardto digitally compress moving picture signals, and has been standardizedby the ISO/IEC (the International StandardizationOrganization/International Engineering Consortium). MPEG-1 is thestandard to compress moving picture signals down to 1.5 Mbps, that is,to compress TV signal information to about one hundredth. Also, thequality which satisfies the MPEG-1 standard is medium level which can berealized at a transmission rate of about 1.5 Mbps. MPEG-2 is thusstandardized in order to satisfy the need for higher picture quality,and it compresses moving picture signals to 2 to 15 Mbps. At present,the work group (ISO/IEC JTC1/SC29/WG11), which standardized MPEG-1 andMPEG-2, has standardized MPEG-4 with a higher compression rate. TheMPEG-4 standard (i) achieves a compression rate higher than those ofMPEG-1 standard and MPEG-2 standard, (ii) enables coding, decoding andperforming operations on an object-by-object basis, and (iii) realizesnew functions necessary in this multimedia era. The initial object ofMPEG-4 standard is to standardize a coding method of pictures with lowbit rates, but the object is extended to a general purpose coding methodof interlace pictures with high bit rates. After that, ISO/IEC andITU-T, in combination, has standardized MPEG-4 AVC (Advanced VideoCoding) as a next generation picture coding method of pictures with ahigh compression rate. This is expected to be used for next generationoptical disc related apparatuses or in broadcasting for mobileterminals.

Generally, in coding moving pictures, information amount is compressedby reducing temporal and spatial redundancies. In the inter pictureprediction coding aiming to reduce temporal redundancies, motionestimation and prediction picture generation are performed on ablock-by-block basis with reference to a forward picture or a backwardpicture, and coding is performed on the differential value between theobtained prediction picture and the picture to be coded. Here, “picture”used here is a term representing one picture. In a progressive picture,a picture means a frame, but in an interlace picture, it means a frameor a field. An “interlace picture” described here means a frame composedof two fields with a slight time lag. In the coding and decodingprocesses of interlace pictures, it is possible to process a frame as itis, as two fields, or on a frame-by-frame or on a field-by-field of eachblock in a frame.

The picture for performing intra prediction coding without referring toany reference picture is called Intra Coded Picture (I picture). Also,the picture for performing inter prediction coding referring to only apicture is called Predictive Coded Picture (P picture). Also, thepicture for performing inter prediction coding referring to tworeference pictures simultaneously is called Bi-predictive Coded Picture(B picture). A B picture can refer to two pictures selected as anarbitrary combination of a forward picture and a backward picture indisplay time. Such two reference pictures can be specified on ablock-by-block basis, the block being a basic unit of coding anddecoding. Those reference pictures are distinguished from each other asfollows: the reference picture described earlier in the coded bit streamis called first reference picture, and the other reference picturedescribed later is called second reference picture. Note that suchreference pictures must have already been coded or decoded in order tocode or decode an I picture, P pictures and B pictures.

Motion compensation intra prediction coding is used for coding of Ppictures and B pictures. Motion compensation intra prediction coding isan intra prediction coding method in which motion compensation isapplied. Motion compensation is a method for improving predictionprecision and reducing data amount by estimating motion amount (calledmotion vector hereafter) of each block of a picture and by performingprediction coding considering the motion vector. For example, dataamount is reduced by estimating motion vectors of pictures to be codedand by coding each prediction residual between each prediction valuewhich is shifted by the amount of each motion vector and each currentpicture to be coded. In the case of this method, since motion vectorinformation is needed in decoding, motion vectors are also coded, andrecorded or transmitted. Motion vectors are estimated on a macro blockby macro block basis. To be more specifically, motion vectors areestimated by fixing the macro block of a picture to be coded, moving themacro block of a reference picture within the search range, and findingthe location of the reference block which is closest to the standardblock.

FIGS. 1A and 1B are structural diagrams of conventional MPEG-2 streamsrespectively.

As shown in FIG. 1B, an MPEG-2 stream has a hierarchical structure likewill be described below. A stream is composed of a Group of Pictures(called GOP hereafter). The use of a GOP as a basic unit in codingprocessing enables editing a moving picture or performing a randomaccess. A GOP is made up of an I picture, P pictures and B pictures. Astream, a GOP and a picture further includes a synchronous signal (sync)indicating a border of units and a header indicating the data common inthe units, the units here being a stream, a GOP and a picturerespectively.

FIGS. 2A and 2B respectively show examples indicating how to performinter picture prediction coding which is used in MPEG-2.

The diagonally-shaded pictures in the figure are those pictures to bereferred to by other pictures. As shown in FIG. 2A, in prediction codingin MPEG-2, P pictures (P0, P6, P9, P12 and P15) can refer to only asingle picture selected as an immediately forward I picture or P picturein display time. Also, B pictures (B1, B2, B4, B5, B7, B8, B10, B11,B13, B14, B16, B17, B19, and B20) can refer to two pictures selected asa combination of an immediately forward I picture or P picture and animmediately backward I picture or P picture. Further, the order ofpictures to be placed in a stream is determined. I pictures and a Ppicture are placed in the order of display time, and each B picture isplaced immediately after an I picture to be displayed immediately afterthe B picture or immediately after a P picture. As a structural exampleof a GOP, as shown in FIG. 2B, pictures from I3 to B14 are grouped intoa single GOP.

FIGS. 3A to 3B show the decoding order, the display order and delayamounts which occur between decoding time and display time of a GOPstructure used in an MPEG-2 stream respectively.

Here, the MPEG-2 stream has a fixed frame rate, and the B pictures aredecoded and displayed simultaneously. In an MPEG-2 stream, as shown inFIGS. 3A and 3B, the delay amount which is the time lag from thedecoding time of the top picture of the GOP to the display time of thetop picture is equivalent to one frame or two fields at maximum. Thisdelay amount will be called frame delay hereafter, and the length of aframe delay will be counted on a frame by frame basis (one framecorresponds to two fields). Optical disc apparatuses such as a DigitalVersatile Disk (DVD) employs the MPEG-2 standard in which it is definedthat frame delays are fixed at one. Note that delay amounts arechangeable at the time of pulling down such as displaying, at 60 Hz, thestreams that have been coded at 24 Hz. Since it is possible to determinethe delay amounts based on the case of displaying the coded streamsaccording to the frame rate, a case of displaying the coded streamsaccording to the frame rate will be described below.

FIG. 4 is a structural diagram of an MPEG-4 AVC stream. There is noconcept equivalent to a GOP in the MPEG-4 AVC. However, since it ispossible to construct a randomly-accessible access unit equivalent to aGOP by segmenting data in a unit of a special picture which can bedecoded without depending on other pictures, the unit will be called RAU(Random Access Unit) hereafter.

There are two types of I pictures in MPEG-4 AVC. They are anInstantaneous Decoder Refresh (IDR) and the rest. An IDR picture is theI picture which can decode all the pictures placed after the IDR picturein a decoding order, without referring to pictures placed before the IDRpicture in the decoding order. An IDR picture corresponds to the top Ipicture of an MPEG-2 closed GOP. In the case of an I picture which isnot an IDR picture, a picture placed after the I picture in the decodingorder may refer to a picture placed before the I picture in the decodingorder. Also, it is possible to form a structure like an open GOP in theMPEG-2 by placing an I picture that is not an IDR picture at the top ofa random access unit RAU and restricting the predictive structure ofpictures in the random access unit RAU.

FIG. 5 is an example of a prediction structure of pictures in an MPEG-4AVC stream.

Since the MPEG-4 AVC allows flexible prediction structures, for example,picture P2 can refer to picture I8. In the example of FIG. 5, sincedisplay is started after picture I8 and picture P2 are decoded first,the frame delay becomes two. Since prediction structures are flexible inthis way, frame delays are not limited to one at maximum like in thecase of MPEG-2. This means that frame delays are variable depending onprediction structures. Therefore, it is impossible to perform playbackon condition that frame delays are fixed at one.

Package media such as DVDs have a special playback function such as (i)playback where particular parts of the same stream are selectivelyplayed back or where different streams can be played back continuouslyand (ii) multi-angle playback where playback is performed changingstreams with a different angle. The basic unit of using such a functionis a GOP in the MPEG-2 and a random access unit RAU in the MPEG-4 AVC.

FIGS. 6A to 6C show an example of changing streams to be played back inthe MPEG-2. FIGS. 6A to 6C respectively show the GOPs included in Stream1, 2 and 3. Here, streams to be played back are changed from Stream 1 toStream 2 by decoding GOP 2-1 next to GOP 1-1. This makes it possible toperform playback in a fixed rate without allowing the occurrence of agap at the time of display because frame delay amounts are one both inthe GOP1-1 and GOP2-1. Likewise, it is possible to change from Stream 1to Stream 3 by decoding GOP3-1 next to GOP1-1.

Conventionally, various techniques relating to moving picture coding,multiplexing, decoding and demultiplexing like those described abovehave been proposed. (For example, refer to Japanese Laid-Open PatentApplication No. 2003-18549 publication.)

FIG. 7 is a flow chart showing the operation of a conventionalmultiplexing apparatus for coding and multiplexing moving picture data.

First, in Step 101 and Step 102, the multiplexing apparatus codes one ormore streams. Next, in Step 103, it generates management information andthen goes to Step 104. Management information includes the informationfor accessing the stream generated in Step 101, the informationindicating data to be played back at the time of special playback suchas multi-angle playback or the like. After that, in Step 104, itmultiplexes the management information with stream data and outputs themultiplexed data.

FIG. 8 is a block diagram showing the structure of a conventionalmultiplexing apparatus.

The multiplexing apparatus 800 includes a coding unit 11, a memory 12, amanagement information generation unit 13 and a multiplexing unit 14.

The coding unit 11 codes the inputted moving picture data Vin and storesthe coded data strIn into the memory 12.

The management information generation unit 13 reads out the coded datafrom the memory 12 as read out data strOut 1, generates managementinformation base and outputs the management information base to themultiplexing unit 14. Note that the management information base does notinclude the information concerning frame delays.

The multiplexing unit 14 multiplexes (i) the management informationbase, (ii) read out data strOut 2 which has been read out from thememory 12, and (iii) addition information adInf such as settinginformation that is set by a user and that is obtained separately fromthe stream, and then outputs the multiplexed data MuxDat. Here, additioninformation adInf may not be used if it is not necessary. Also, the readout data strOut 2 may be packetized using a scheme such as MPEG-2Transport Streams (TSs) or Program Streams (PSs), or other schemepredetermined by application, and then multiplexed. For example, in theBlu-ray Disc (BD) standard, the read out data strOut 2 is multiplexedusing a scheme where 4-byte header is added to MPEG-2 TS packets calledSource Packets, and then stored.

FIG. 9A shows the structural example of the multiplexed data outputtedfrom the multiplexing apparatus 800.

As shown in FIG. 9A, management information and one or more coded streamare stored in the multiplexed data. Further, handling each stream as oneor more clips makes it possible to realize various playback methods suchas digest playback and multi-angle playback. Here, a clip shows onepicture or a sequence of pictures in one random access unit RAU or asequence of random access unit RAUs of the same stream, and the clip andthe stream may be the same. FIGS. 9B and 9C show playback examples.Especially, FIG. 9B shows an example of multi-angle playback. In thecase where Stream 1 and Stream N respectively store video at a differentangle, it is possible to play back Clip N-2 of Stream N by changingangles next to Clip 1-1 of Stream 1 and return to the playback of Stream1 after completing the playback of Clip N-2. FIG. 9C shows an example ofdigest playback. It is possible, for example, to play back typicalscenes by selectively playing back Clip 1-1 and Clip 1-M in Stream 1.

FIG. 10 is a flow chart showing the operation of a conventionaldemultiplexing apparatus for demultiplexing the multiplexed data toobtain the coded data and playing back the coded data.

First, in Step S201, the demultiplexing apparatus demultiplexes themultiplexed data to obtain management information, obtains theinformation concerning the one or more clips to be played back, and thengoes to Step 204. The information concerning clips includes start timeor end time of the clips, access information used for accessing thecoded data in the clips and the like. In Step 204 and Step 205, thedemultiplexing apparatus decodes and displays pictures in the clips upto the last pictures in the clips. Here, in the case where aninstruction indicating the completion of playback is made by useroperation or the like, the playback is completed at the time when theinstruction becomes valid.

FIG. 11 is a block diagram showing the structure of a conventionaldemultiplexing apparatus 900.

The demultiplexing apparatus 900 includes a management informationdemultiplexing unit 21, a clip information analysis unit 22, a decodingunit 24 and a display unit 26.

The management information demultiplexing unit 21 reads out multiplexeddata MuxDat from a multiplexed data recording medium such as an opticaldisc, analyzes the management information, and determines clips to beplayed back according to the user instruction or a predetermined method.After that the management information demultiplexing unit 21 outputs, tothe clip information analysis unit 22, the clip information Clip that isthe information concerning the determined clips.

The clip information analysis unit 22 outputs, to the decoding unit 24,access information acs used for accessing the pictures that constitutethe clips. On the other hand, the decoding unit 24 reads out the videodata Vdat from the multiplexed data recording medium based on the accessinformation acs, decodes the read-out data, and outputs the decodingresult decOut to the display unit 26. The display unit 26 displays thedecoding results in the display order.

The MPEG-4 AVC allows flexible prediction structures, and thus framedelays of clips are variable. Since a conventional demultiplexingapparatus changes clips without considering frame delays of clips, a gapin a display interval of pictures occurs at the time of changing clipswith a different frame delay.

FIGS. 12A to 12C show an example of changing from the clip withone-frame delay to the frame with two-frame delay.

FIG. 12A shows the random access unit RAU1-1 of Stream 1 with one-framedelay, while FIG. 12B shows the random access unit RAU2-1 of Stream 2with two-frame delay. Here, FIG. 12C shows the timing of decoding anddisplaying at the time of playing back the RAU2-1 next to the RAU1-1.

Since the frame delay of RAU1-1 is one, at the time when picture P15that is the last in the decoding order of RAU1-1 is displayed, pictureI8 that is the top picture of RAU2-1 is decoded. However, the framedelay of RAU2-1 is two, at the time when picture P2 that is the secondin the decoding order is decoded, display of the pictures in the RAU2-1has yet to be started. Therefore, there is no picture to be displayed atthe time when picture P2 is decoded. Consequently, a gap in a displayinterval occurs between picture P15 and picture B0.

Likewise, in the case of playing back the random access unit RAU1-1after the random access unit RAU2-1, a gap occurs in the decodinginterval to display pictures continuously. In other words, there occursan overlap in the display interval. A gap in the display order means,hereinafter, discontinuity at a connection that occurs in both caseswhere frame delay amount at a connection increases and decreases.

As described up to this point, conventional multiplexing anddemultiplexing apparatuses have a problem of making a user who watchesthe moving picture feel uncomfortable because the conventionaldemultiplexing apparatus cannot display pictures placed at the part atwhich clips with a different frame delay are changed, maintaining afixed frame.

The present invention is conceived in order to solve the above-describedproblem. An object of the present invention is to provide a multiplexingapparatus for multiplexing the coded stream with other information so asto generate multiplexed data and a demultiplexing apparatus fordemultiplexing the multiplexed data to play back the coded stream sothat they do not make the user feel uncomfortable even at the time ofperforming any special playback such as multi-angle playback.

DISCLOSURE OF INVENTION

In order to achieve the above-described object, the multiplexingapparatus, of the present invention, multiplexes one or more codedstreams with other information, the coded streams including codedpictures in a unit of a randomly-accessible access unit, themultiplexing apparatus including: a coding unit for generating the codedstreams so that no gap occurs at a connection of predetermined twoaccess units in a process of decoding, in sequence, the two access unitsin all of access units included in the coded streams; and a multiplexingunit for multiplexing, with the other information, the coded streamsgenerated by the coding unit. For example, in a first aspect of thepresent invention, the coding unit generates the coded streams so thatdelay amounts in the two access units become equal to each other, eachof the delay amounts being a time lag from when a top picture in adecoding order is decoded to when a top picture in a display order isdisplayed.

In this way, coded streams are generated so that delay amounts (framedelays) of two access units (such as clips) on which special playback isperformed become equal to each other. Therefore, no gap occurs at theconnection of these access units at the time when these access units aredecoded in sequence. In other words, these access units are connected toeach other seamlessly at the time of special playback. This means theframe rates become invariable. Consequently, it becomes possible toeliminate making a user who watches a moving picture made up of theseaccess units feel uncomfortable. Here, for example, coded streams aregenerated so that the delay amounts of these two access units areadjusted to predetermined delay amounts in the case where delay amountsare predetermined according to the operation standard of an applicationprogram.

Also, in a second aspect of the present invention, the multiplexingapparatus may further include a delay information generation unit forgenerating delay information concerning the delay amounts, in theapparatus, the multiplexing unit may multiplex, with the otherinformation, the delay information generated by the delay informationgeneration unit. For example, in a third aspect of the presentinvention, the delay information generation unit generates the delayinformation indicating the delay amounts. Otherwise, in the third aspectof the present invention, the delay information generation unitgenerates, as the delay information, a flag indicating that the twoaccess units can be connected seamlessly.

In this way, the demultiplexing apparatus can easily obtain the delayamounts in these two access units through the delay information in thecase where the delay information (frame delay information) indicatesdelay amounts. Also, the demultiplexing apparatus can easily obtain thedelay amounts in these two access units in the case where a flag asdelay information indicates that the delay amounts of these two accessunits are equal to each other. Consequently, it is possible to cause thedemultiplexing apparatus to perform more proper demultiplexingprocessing.

Also, in the second aspect of the present invention, the coding unit maygenerate the coded streams so that no gap occurs at a connection of thetwo access units in all of the access units included in the codedstreams, the connection being an angle changeable point.

Since these two access units have a different angle, no gap occurs atthe connection of these access units at the time when these access unitsare decoded in sequence likewise the earlier described case. In otherwords, these access units are connected in a seamless multi-anglemanner. This means that the frame rates are invariable. Consequently, itbecomes possible to eliminate making a user who watches a moving picturemade up of these access units with a different angle feel uncomfortable.

Also, in order to achieve the above-described object, the demultiplexingapparatus, of the present invention, demultiplexes multiplexed data, themultiplexed data including: (i) coded streams including coded picturesin a unit of a randomly-accessible access unit; and (ii) delayinformation concerning delay amounts in predetermined two access unitsin all of access units, each of the delay amounts being a time lag fromwhen a top picture in a decoding order is decoded to when a top picturein a display order is displayed, and the demultiplexing apparatusincluding: a delay information demultiplexing unit for demultiplexingthe multiplexed data to obtain the delay information; and a playbackunit for decoding and playing back the access units in sequenceaccording to the delay information obtained when the delay informationdemultiplexing unit has demultiplexed the multiplexed data.

In an example case where coded streams are generated so that delayamounts of these two access units (such as clips) on which specialplayback is performed, it becomes possible to decode and play back thesetwo access units in sequence without allowing the occurrence of a gap atthe connection of these two access units at the time of specialplayback. In other words, it becomes possible to connect these accessunits seamlessly. This means that it becomes possible to make the framerates invariable. Consequently, it also becomes possible to eliminatemaking a user who watches the moving picture made up of these accessunits feel uncomfortable. Also, in an example case where delay amountsare indicated by delay information (frame delay information), it ispossible to easily obtain the delay amounts of these two access unitsthrough the delay information. Further, in the case where the delayinformation is indicated using a flag, it is possible to easily obtainthe information indicating that the delay amounts of these two accessunits are equal to each other. Consequently, it is possible to performmore proper demultiplexing processing.

Also, in order to achieve the above-described object, in themultiplexing method of the present invention, a moving picture is codedand the coded moving picture is multiplexed with management information,the multiplexing method including: generating one or more coded streams;obtaining frame delays in the coded streams; generating managementinformation including the information indicating the obtained framedelays; and multiplexing the coded streams with the managementinformation. In the multiplexing method, the coded streams are made upof one or more random access units, the frame delays indicate the delayamounts that occur respectively during the time periods from when thetop picture in the decoding order is decoded to when the top picture inthe display order is displayed in these random access units, and theframe delays in the coded streams are variable.

Also, the management information may include the frame delays of thecoded streams.

Also, the management information may include the maximum value of theframe delays in the coded streams.

Also, the management information may include the equal frame delays ascommon frame delays in the coded streams at the time when the framedelays are equal to each other.

Also, the management information may include the frame delays based on aplayback unit made up of these random access units.

Further, the demultiplexing method of the present invention is thedemultiplexing method for demultiplexing the data multiplexed accordingto the multiplexing method and playing back the resulting demultiplexeddata. The demultiplexing method includes: obtaining the frame delays ofthe coded streams to which these random access units to be played backbelong; determining the frame delays at the time of playback based onthe obtained frame delays; and playing back pictures included in theserandom access units according to the determined frame delays.

Also, in the determination of frame delays, these random access unitsthat follow the current random access unit may be played back with theframe delays equal to the frame delay of the random access unit that hasjust been played back as the current random access unit.

Note that the present invention can be realized not only as themultiplexing apparatus, the demultiplexing apparatus, the multiplexingmethod and the demultiplexing method that have been described above butalso a program for the multiplexing apparatus and the demultiplexingapparatus, a recording medium for storing the program and themultiplexed data generated by the multiplexed apparatus.

Up to this point, since the present invention makes it possible toguarantee display with a fixed frame rate even at the time of specialplayback, it is possible to eliminate making a user feel uncomfortable.Especially, the present invention makes it possible to improve theplayback quality of a package medium, and thus it is highly practical.

FURTHER INFORMATION ABOUT TECHNICAL BACKGROUND TO THIS APPLICATION

The disclosure of Japanese Patent Application No. 2004-165030 filed onJun. 2, 2004 including specification, drawings and claims isincorporated herein by reference in its entirety.

BRIEF DESCRIPTION OF DRAWINGS

These and other objects, advantages and features of the invention willbecome apparent from the following description thereof taken inconjunction with the accompanying drawings that illustrate a specificembodiment of the invention. In the Drawings:

FIGS. 1A and 1B are illustrations of the structure of an MPEG-2 stream;

FIGS. 2A and 2B are illustrations of the structures of a GOP in theMPEG-2 stream;

FIGS. 3A and 3B are illustrations of frame delays in MPEG-2 streams;

FIG. 4 is a diagram showing the structure of an MPEG-4 AVC stream;

FIG. 5 is a diagram showing an example of a prediction structure in theMPEG-4 AVC stream;

FIGS. 6A to 6C are diagrams showing examples of how clips are changed inMPEG-4 AVC streams;

FIG. 7 is a flow chart showing the operation of a conventionalmultiplexing apparatus;

FIG. 8 is a block diagram showing the structure of the conventionalmultiplexing apparatus;

FIGS. 9A to 9C are diagrams showing structure examples of the dataoutputted from the conventional multiplexing apparatus;

FIG. 10 is a flow chart showing the operation of the conventionalmultiplexing apparatus;

FIG. 11 is a block diagram showing the structure of the conventionalmultiplexing apparatus;

FIGS. 12A to 12C are diagrams showing examples of problems that occur atthe time of playing back the multiplexed data outputted by theconventional multiplexing apparatus;

FIG. 13 is a flow chart showing the operation of the multiplexingapparatus in a first embodiment of the present invention;

FIG. 14 is a block diagram showing the multiplexing apparatus in thefirst embodiment of the present invention;

FIGS. 15A to 15E are diagrams showing examples of data structuresoutputted by the multiplexing apparatus in the first embodiment of thepresent invention;

FIG. 16 is a flow chart showing the operation of the multiplexingapparatus in a second embodiment of the present invention;

FIG. 17 is a block diagram of the multiplexing apparatus in the secondembodiment of the present invention;

FIG. 18 is a flow chart showing the operation of the demultiplexingapparatus in a third embodiment of the present invention;

FIG. 19 is a flow chart showing the operation of the demultiplexingapparatus at the time of determining a frame delay in playback in thethird embodiment of the present invention;

FIGS. 20A to 20D are diagrams showing the operation examples of thedemultiplexing apparatus at the time of determining a frame delay inplayback in the third embodiment of the present invention;

FIG. 21 is a block diagram showing the demultiplexing apparatus in thethird embodiment of the present invention;

FIG. 22 is a diagram showing the data hierarchy of an HD-DVD;

FIG. 23 is a diagram showing the structure of the logical space of anHD-DVD;

FIG. 24 is a diagram showing the structure a VOB information file;

FIG. 25 is an illustration of a time map;

FIG. 26 is a diagram showing the structure of a play list file;

FIG. 27 is a diagram showing the structure of a program filecorresponding to a play list;

FIG. 28 is a diagram showing the structure of a BD disc managementinformation file;

FIG. 29 is a diagram showing the structure of a program file forrecording a global event handler;

FIG. 30 is a block diagram showing the outline structure of an HD-DVDplayer; and

FIGS. 31A and 31B are illustrations showing the structure of a recordingmedium on which the program for realizing the multiplexing method andthe demultiplexing method of the present invention are recorded; andFIG. 31C is an illustration of the structure for recording and playingback the program on the flexible disc

BEST MODE FOR CARRYING OUT THE INVENTION

Embodiment of the present invention will be described below withreference to figures.

First Embodiment

FIG. 13 is a flow chart showing the operation of the multiplexingapparatus in a first embodiment of the present invention. Themultiplexing apparatus outputs multiplexed data that can indicate framedelay information of the coded stream to be stored in the multiplexeddata.

First, in Step 301, the multiplex apparatus generates MPEG-4 AVC codedstreams. In Step 302, it obtains frame delays of the coded streamgenerated in Step 301, and goes to Step 303. In Step 303, it judgeswhether or not all the coded streams to be stored in the multiplexeddata have already been generated, and repeats processing of Step 301 andStep 302 until all the coded streams have been generated. In Step 304,it generates frame delay information to be stored as managementinformation of multiplexed data, and goes to Step 305. In Step 305, itgenerates management information indicating frame delay information inaddition to the information generated by a conventional multiplexingapparatus. Lastly, in Step 306, it multiplexes the coded streamsgenerated in Step 301 with the management information generated in Step305, and then it outputs the multiplexed data.

FIG. 14 is a block diagram showing the structure of the multiplexingapparatus in the first embodiment of the present invention.

The multiplexing apparatus 100A includes a coding unit 15, a memory 12,a management information generation unit 16, a frame delay obtainmentunit 17 and a multiplexing unit 18.

Note that the same components of the above-described components of themultiplexing apparatus 100A of this embodiment as the conventionalmultiplexing apparatus shown in FIG. 8 are assigned the same referencenumbers, and such components will not be described here again.

The coding unit 15 codes the inputted moving picture data Vin, andstores the resulting coded data strIn into the memory 12. Also, itoutputs, to the frame delay obtainment unit 17, frame delay frDly of thecoded data strIn.

The frame delay obtainment unit 17 generates delay information dlyInf tobe stored in the management information as the frame delay information,and then outputs it to the management information generation unit 16.

The management information generation unit 16 generates managementinformation base based on the analysis result of the coded data strInread out from the memory 12 as read-out data strOut 1 and the delayinformation dlyInf, and then outputs it to the multiplexing unit 18.

The multiplexing unit 18 multiplexes the management information base,the read out data strOut 2 read out from the memory 12 and additioninformation adInf, being setting information by user, to be obtainedseparately from the coded data, and then outputs the multiplexed dataMuxDat. Note that the frame delay at the time of coding may be limitedto a predetermined value or below, the frame delay being set by thecoding unit 15.

FIGS. 15A to 15E show the structural example of the multiplexed dataoutputted by the multiplexing apparatus 100A.

The multiplexed data shows frame delay information of clips to be storedin the multiplexed data as shown in FIG. 15A. Here, N numbers of clipsare stored in the multiplexed data, and frame delays are from delay 1 todelay N. Note that respective clips may be the unit indicating differentareas in the same coded stream, or may be the clips that belong todifferent coded streams.

FIGS. 15B to 15D show examples of frame delay information. FIG. 15Bstores frame delays of respective clips as table information. FIG. 15Cshows the maximum value of frame delays of the clips. As the maximumvalue, the maximum value among the frame delays of all the clips to bestored in the multiplexed data may be shown, or the maximum value amongthe frame delays of the clips to be played back in sequence may be shownbased on a play list. Also, a predetermined value may be shown as themaximum value. FIG. 15D shows the value of frame delay to be used incommon among the respective clips. In the case where frame delays of therespective clips are variable, these values may be shown. Also, in thecase where frame delays of the respective clips are invariable, framedelays used at the time of playback may be shown. FIG. 15E shows whetheror not frame delays of the clips are equal to each other. For example,flag information indicating whether or not these frame delays are equalto each other is stored. Note that the information shown in FIGS. 15B to15D may be used in combination.

Note that frame delay information concerning only particular clips maybe shown. First, it is possible to show the frame delay informationconcerning clips to be used for multi-angle playback or digest playbackbased on the playback method of clips. Also, it is possible to set framedelay based on the attribute of the top random access unit in a clip.For example, in the case where it is defined that angles are changed atan IDR picture, frame delay information only concerning a clip whose toprandom access unit is an IDR picture may be shown. Also, frame delayinformation may be shown only in the case of digest playback in which itis guaranteed that clips are connected to each other seamlessly.

Also, frame delay information of the coded streams to be stored in themultiplexed data may be shown without directly showing the frame delayinformation of the respective clips. At this time, it is possible toshow frame delays of the respective clips by associating the codedstream to which each clip belong with the information indicating theframe delays of the coded stream. This method may be used both (i) inthe case where frame delays of the respective clips in the same codedstream are invariable and (ii) in the case where the maximum value offrame delays of the clips included in the same coded stream is shown.

In the case of using an application standard, such as the Blu-Ray Disc(BD) or a High Definition (HD)-DVD, for which it is defined that framedelays should be equal to each other or for which the maximum value or apredetermined value among frame delays is defined, it should be notedthat frame delays may not be stored as management information becausesuch an application standard shows the information concerning the framedelays.

Also, in the case of receiving multiplexed data via a network by using aprotocol such as TS or Real-time Transmission Protocol (RTP), framedelay information may be obtained as playback control information. Forexample, in the case of using the Session Description Protocol (SDP) atthe time of notifying a playback terminal the playback controlinformation, it is possible to describe the frame delay information inthe SDP. Also, frame delays may be notified to the playback terminal byindicating frame delay information in a scene description language suchas a Synchronized Multimedia Integration Language (SMIL).

Also, frame delay information may be indicated on a random access byrandom access basis. Further, frame delay information may be indicatedin the coded stream by, for example, adding it to the top picture in arandom access unit RAU.

Frame delay information is indicated by management information in themultiplexed data to be outputted by the multiplexing apparatus in thisway. Therefore, adjusting frame delays at the time of playing back themultiplexing data makes it possible to play back the coded streamwithout allowing the occurrence of a gap in display at the time ofchanging clips.

Second Embodiment

FIG. 16 is a flow chart showing the operation of the multiplexingapparatus in a second embodiment of the present invention.

First, in Step 401, the multiplexing apparatus sets the value of framedelay used in common in the coded stream to be stored in the multiplexeddata. In Step 402, it generates MPEG-4 AVC coded streams based on theframe delay that has been set in Step 401. In Step 403, it judgeswhether all the coded streams to be stored in the multiplexed data havealready been generated, and repeats the processing of Step 402 until allthe coded streams are coded. In Step 404, it generates frame delayinformation to be stored as management information of the multiplexeddata, and then goes to Step 405. In Step 405, it generates managementinformation indicating frame delay information in addition to theinformation generated by the conventional multiplexing apparatus.Lastly, in Step 406, it multiplexes the coded streams generated in Step401 and the management information generated in Step 405 and thenoutputs the resulting multiplexed data.

FIG. 17 is a block diagram showing the structure of the multiplexingapparatus in the second embodiment of the present invention.

The multiplexing apparatus 100B includes a frame delay determinationunit 41, a coding unit 42, a memory 12, a management informationgeneration unit 16 and a multiplexing unit 18.

Note that the same components of the above-described components of themultiplexing apparatus 100B of this embodiment as the conventionalmultiplexing apparatus shown in FIG. 8 are assigned the same referencenumbers, and such components will not be described here again.

The frame delay determination unit 41 determines the frame delay of thecoded stream and outputs delay information dlyInf to the coding unit 42and the management information generation unit 16. The frame delaydetermined here may be the value predetermined by an applicationstandard, or may be set for the multiplexing apparatus or by a user.

The coding unit 42 codes the inputted moving picture data Vin based onthe frame delay indicated by delay information dlyInf, and then storesthe coded data strIn into the memory 12. In other words, the coding unit42 like this generates one or more coded data strIn making the framedelays in a unit of a random access unit equal to each other so that nogap occurs at the connection of these access units at the time whenparticular two access units, which are access units (coded datathemselves or clips) included in one or more coded data strIn, aredecoded in sequence. Note that to prevent the occurrence of a gap at theconnection of these access units at the time of decoding means toprevent the occurrence of a gap in a display interval of pictures andprevent the occurrence of an overlap in a decoding interval of pictures.

The management information generation unit 16 generates managementinformation base based on the analysis result of the coded data strOut1read out from the memory 12 and delay information dlyInf, and thenoutputs the management information base to the multiplexing unit 18.

The multiplexing unit 18 multiplexes (i) management base, (ii) read outdata strOut2 that has been read out from the memory 12 and (iii)addition information adInf, such as setting information by a user, whichis obtained separately from the coded data, and then outputs theresulting multiplexed data MuxDat.

Note that, in the case where frame delay is previously defined by anapplication standard or the like, the multiplexing apparatus 100B mayhave the structure from which the frame delay determination unit 41 isexcluded, and the coding unit 42 may perform coding processing based ona fixed frame delay instead. On condition that frame delays areinvariable in a coded stream, it is possible to determine a frame delayat the time of playback without frame delay information included inmanagement information. Therefore, there is no need to always indicateframe delay information in management information.

Since frame delays in the coded stream to be stored in the multiplexeddata are invariable in this way in the case of multiplexed dataoutputted by the multiplexing apparatus, frame delays of respectiveclips are also invariable. Therefore, it is possible to play back codedstreams with no gap in display without adjusting frame delays at thetime of starting playback even in the case where clips are changed.

Note that a coding scheme other than the MPEG-4 AVC may be used in theabove-described embodiments as long as it defines that frame delays arevariable.

Also, the frame delays of these random access units (coded streams orclips) to be played back in sequence in seamless connection or seamlessmulti-angle playback may be made invariable. Here, seamless connectionmeans to connect clips included in the same stream or different streamsseamlessly. Also, seamless multi-angle playback means to connect clips,whose angles are different from each other, included in the codedstreams and change these angles. For example, playback may be performedusing an angle during the time period of the starting time up to 30^(th)second and selecting plural angles during the time period of 30^(th)second to 60^(th) second. At this time, each of the clips that shows adifferent angle has the same frame delay. Further, frame delay per anglemay be made invariable irrespective of whether it is possible to performmulti-angle playback seamlessly. This is because the reproductionquality does not become invariable in the case where frame delays varyin a unit of an angle at the time of multi-angle playback. Note thatsuch seamless connection or seamless multi-angle playback may beperformed on the coded streams.

Note that each access unit used in seamless connection or multi-angleplayback may be required to satisfy the following condition: anarbitrary picture in a first access unit can be decoded without thereference to a picture in a second access unit decoded earlier than thefirst access unit. An access unit like this corresponds to a closed GOPin the MPEG-2 or a random access unit RAU that starts with an IDRpicture in the MPEG-4 AVC. Therefore, frame delays in the access unitswhose constituent pictures can be decoded referring to a picture withinthe same access units may be made invariable.

Also, in the case where frame delays of these access units to be playedback in sequence in seamless connection or the like that are describedabove are made invariable, the multiplexing apparatus previously judgeswhether seamless connection or seamless multi-angle playback should beperformed on these access units that should be generated. After that, inthe case where the multiplexing apparatus judges that these access unitsshould be the target, it performs coding so that frame delays of thetarget random access units are made to be equal to each other so as togenerate a coded stream. For example, in the case where seamlessconnection or the like is performed on two coded streams, it generatesthe backward coded stream so that frame delays of the backward codedstream are made to be equal to the frame delays of the forward codedstream.

Also, the frame delay information included in the management informationbase may indicate that frame delays to particular access units (codedstream or clips) such as seamless connection are common. Also, it may bea flag indicating that the access unit may become a target of seamlessconnection, multi-angle playback or the like. The flag like this isstored in a play list, a time map or the like as the informationindicating the attribute of connection.

Third Embodiment

FIG. 18 is a flow chart indicating the operation of the demultiplexingapparatus in a third embodiment of the present invention. Thedemultiplexing apparatus inputs and plays back the multiplexed datagenerated by the demultiplexing apparatus in the first and secondembodiments.

First, in Step 501, the demultiplexing apparatus demultiplexes themultiplexed data to obtain the management information, and obtains theinformation concerning the clips to be played back. Clips to be playedback are determined based on the playback order predetermined by aninstruction from a user or in a play list in the management informationor the like.

Next, in Step 502, it obtains frame delay information of the clipsdetermined as the clips to be played back in Step 501, and goes to Step503. In Step 503, it determines the frame delay at the time of playingback the clips based on the frame delay information obtained in Step502.

In Step 504 and Step 505, it decodes and displays the pictures in theclips up to the last pictures in the clips. Here, in the case where thecompletion of the playback is instructed through user operation, theplayback is completed at the time when the instruction becomes valid.Note that frame delays may be determined only at the time of changingclips that belong to different coded streams, in Step 501 to Step 503.

FIG. 19 is a flow chart showing the operation at the time of determiningframe delays during playback in Step 503 of FIG. 18.

First, in Step 601, the demultiplexing apparatus judges whether or notframe delays of the coded streams stored in the demultiplexed data areinvariable. In the case where frame delays are invariable, it goes toStep 603. In contrast, in the case where frame delays are variable, itgoes to Step 602.

In Step 603, it determines to play back the coded streams in themultiplexed data with a common frame delay. In Step 602, it judgeswhether or not the frame delays of the clips to be played back insequence are invariable. In the case where frame delays are invariable,it goes to Step 604. In contrast, in the case where frame delays arevariable, it goes to Step 605. In Step 604, it determines to play backthe coded streams based on the frame delay of the top clip in playback.In Step 605, it determines to adjust frame delays in playback. Availablemethods for adjusting frame delays in playback will be described below.

1. Adjusting to the maximum frame delay among the frame delays of clipsto be played back.

2. Adjusting to the frame delay of the clip played back immediatelybefore.

3. Using predetermined frame delay.

The first method is especially effective in the case where clips to beplayed back can be predetermined. The second method is especiallyeffective in the case where clips to be played back are dynamicallychanged by a user instruction. Also, the third method is effective inthe case where the maximum value of frame delays can be obtained fromthe management information of multiplexed data or the information in thecoded stream or by an application standard. Further, frame delayspredetermined for the apparatus may be used.

FIGS. 20C and 20D show the examples of the first and second methodsrespectively. FIG. 20C shows the example of playing back Clip 2 with atwo-frame delay next to Clip 1 with a one-frame delay. Here, the framedelay at the time of playing back Clip 1 is two. FIG. 20D shows theexample where changing to Clip 1 with a one-frame delay is instructed bya user while Clip 2 with a two-frame delay is being played back. At thistime, the frame delay of Clip 1 is originally one, but it is played backon the premise that the frame delay of Clip 1 is two according to theframe delay of Clip 2. Determining the frame delay in this way makes itpossible to play back a coded stream without allowing the occurrence ofa gap in a display interval at the position where changing between Clip1 and Clip 2 is performed.

Note that there may be a case where frame delay information is not shownin the demultiplexing data when frame delays of the coded stream inmultiplexed data are equal to each other, as shown in the secondembodiment. The processing of Step 502 is unnecessary in the case ofplaying back multiplexed data like this. Also, it is preferable toadjust to the frame delay of the top clip in playback also in the caseof determining the frame delay at the time of starting display in Step503.

Also, the demultiplexing apparatus may always play back the multiplexeddata based on the predefined maximum value in the case where the maximumvalue of frame delays is defined by an application standard or the like.

FIG. 21 is a block diagram showing the structure of the demultiplexingapparatus of a third embodiment.

The demultiplexing apparatus 200 includes a management informationdemultiplexing unit 51, a clip information analysis unit 52, a framedelay determination unit 53, a decoding unit 24 and a display unit 54.

The management information demultiplexing unit 51 reads out multiplexeddata MuxDat from a multiplexed data recording medium such as an opticaldisc, analyzes the management information and determines the clips to beplayed back according to a user instruction or a predetermined method.After that, the management information demultiplexing unit 51 outputsclip information Clip that is the information concerning the determinedclips, to the clip information analysis unit 52.

The clip information analysis unit 52 outputs access information acsused for accessing the pictures that constitute clips to the decodingunit 24. Further, the clip information analysis unit 52 obtains delayinformation dly of the clips to be played back and outputs it to theframe delay determination unit 53.

The decoding unit 24 reads out video data Vdat from a multiplexed datarecording medium based on the access information acs, decodes the videodata Vdat, and then outputs the decoding result decOut to the displayunit 54.

The frame delay determination unit 53 determines frame delay at the timeof playback, and outputs delay T to the display unit 54.

The display unit 54 displays the pictures according to delay T. Notethat there may be a case where frame delay information is not shown inthe multiplexed data in the case where frame delays of the coded streamin the multiplexed data are equal to each other, as shown in the secondembodiment. The demultiplexing apparatus may have the structure fromwhich the frame delay determination unit 53 is excluded in the case ofplaying back multiplexed data like this.

Also, a frame delay at a connection may be variable in the case whereseamless connection is not guaranteed although a frame delay of acontinuous playback unit shown in a play list or the like is invariableat the time when seamless connection or multi-angle playback isperformed on these access units. In the case where managementinformation of multiplexed data does not include information indicatinga frame delay at this time, the frame delay of the access unit may beobtained in the decoding unit 24 and the obtained frame delay may beinputted into the frame delay determination unit 53. Here, in theplayback area where frame delay is invariable, the frame delay isdetermined according to the frame delay of the top access unit in aplayback order. In contrast, in the playback area where frame delay isvariable, the frame delay determination unit adjusts the frame delay atthe time of playback. Note that it is possible to apply the same framedelay as the maximum value of the frame delay determined in anapplication standard or the like in any above-described cases.

Fourth Embodiment

Functions such as multi-angle playback and digest playback areespecially important to optical disc apparatuses for playing backpackage media. Here will be described an example where multiplexed dataoutputted from the multiplexing apparatuses in the first and secondembodiments are recorded into a Blu-ray Disc (BD) that is a nextgeneration optical disc.

First, a recording format of a BD-ROM will be described.

FIG. 22 is a diagram indicating the structure of the BD-ROM, especiallythe structures of a BD disc 104 that is a disc medium, and data storedin the disc. The data stored in the BD disc 114 includes AV data 103, BDmanagement information 102 such as management information concerning theAV data and an AV playback sequence, and a BD playback program 101 thatrealizes interactivity. Here, as a matter of convenience, thedescription of the BD disc will be made focusing on the AV applicationfor playing back audio and visual contents of movies, but similardescription can be made focusing on other use.

FIG. 23 is a diagram showing the structure of a directory file oflogical data stored in the above-described BD disc. A BD disc has arecording area from its inner radius to its outer radius like, forexample, a DVD, a CD and the like, and has logical address space forstoring logical data between the read-in at the inner radius and theread-out at the outer radius. Also, inside the read-in, there is aspecial area that can be read out only by a drive called a Burst CuttingArea (BCA). As this area cannot be read out from application, it may beused for, for example, copy right protection technique.

File system information (volume) is stored in the top of the logicaladdress space, and application data such as video data is also storedthere. As described in the background art, a file system is, forexample, the UDF or the ISO9660, and it enables reading out the logicaldata stored using a directory structure or a file structure like in thecase of a normal PC.

In this embodiment, as the directory structure and the file structure onthe BD disc, the BDVIDEO directory is placed immediately below a routedirectory (ROOT). This directory is a directory storing data such as AVcontents or management information (101, 102 and 103 that are describedin FIG. 22) that is handled in the BD.

Below the BDVIDEO directory, the following seven files are recorded.

(i) BD.INFO (the file name is fixed) which is an item of “BD managementinformation” and is a file storing the information concerning the wholeBD disc. The BD player reads out this file first.

(ii) BD.PROG (the file name is fixed) which is one of “BD playbackprograms” and is a file storing the playback control informationconcerning the whole BD disc.

(iii) XXX.PL (“XXX” is variable, and the extension “PL” is fixed) whichis an item of “BD management information” and is a file storing the playlist information that is a scenario (playback sequence). Each play listhas a file.

(iv) XXX.PROG (“XXX” is variable, and the extension “PROG” is fixed)which is one of “BD playback programs” and is a file storing theplayback control information prepared on the play list basis. Thecorresponding play list is identified based on a file body name (basedon a matching of “XXX”).

(v) YYY.VOB (“YYY” is variable, and the extension “VOB” is fixed) whichis one of “AV data” and is a file storing the VOB (the same as the VOBdescribed in the background art). Each VOB has a file.

(vi) YYY.VOBI (“YYY” is variable, and the extension “VOBI” is fixed)which is an item of “BD management information” and is a file storingthe stream management information concerning the VOB that is the AVdata. The corresponding play list is identified based on a file bodyname (based on a matching of “YYY”).

(vii) ZZZ.PNG (“ZZZ” is variable, and the extension “PNG” is fixed)which is one of “AV data” and is a file storing image data PNG (that isa picture format standardized by the W3C and called “ping”) forconstituting subtitles and menus. Each PNG image has a file.

The structure of BD navigation data (BD management information) will bedescribed with reference to FIG. 24 to FIG. 29.

FIG. 24 is a diagram showing the internal structure of a VOB managementinformation file (“YYY.VOBI”).

The VOB management information has the stream attribute information(Attribute) of the VOB and a time map (TMAP). The stream attribute hasvideo attribute (Video) and audio attribute (Audio#0 to Audio#m)separately. Especially in the case of audio stream, as a VOB has pluralaudio streams at the same time, the presence or absence of a data fieldis indicated by the number (Number) of audio streams.

The following are video attributes (Video) stored in fields respectivelyand the values that the respective fields may have.

(i) compression format (Coding): MPEG-1; MPEG-2; MPEG-4; and MPEG-4 AVC(Advanced Video Coding).

(ii) resolution (Resolution): 1920×1080; 1440×1080; 1280×720; 720×480;and 720×565.

(iii) aspect ratio (Aspect): 4 to 3; and 16 to 9.

(iv) frame rate (Framerate): 60; 59.94 (60/1.001); 50; 30; 29.97(30/1.001); 25; 24; and 23.976 (24/1.001).

The following are audio attributes (Audio) stored in fields respectivelyand the values that the respective fields may have.

(i) compression format (Coding): AC3; MPEG-1; MPEG-2; and LPCM.

(ii) the number of channels (Ch): 1 to 8

(iii) language attribute (Language):

The time map (TMAP) is a table for storing the information on a VOBUbasis, and has the number of VOBUs that the VOB has and the respectiveitems of VOBU information (VOBU#1 to VOBU#n). The respective items ofVOBU information include I_start that is the address (the startingaddress of an I picture) of the top TS packet of a VOBU and an offsetaddress (I_end) up to the end address of the I picture, and the playbackstarting time (PTS) of the I picture.

FIG. 25 is a diagram illustrating the details of the VOBU information.

As widely known, as variable bit rate compression may be performed onthe MPEG video stream in order to record the video stream in highquality, there is no proportionality between the playback time and thedata size. On the other hand, as a fixed bit rate compression isperformed in the AC3 that is an audio compression standard, therelationship between the time and the address can be obtained from aprimary expression. However, in the case of MPEG video data, each framehas a fixed display time, for example, a frame has display time of1/29.97 seconds in the case of NTSC, but the data size after compressingeach frame changes greatly depending on the image feature, or thepicture type used in the compression, such as an I picture, a P pictureor a B picture. Therefore, in the case of an MPEG video stream, it isimpossible to represent the relationship between the time and theaddress using a primary expression.

As might be expected, it is impossible to represent the relationshipbetween the time and the data size using a primary expression in an MPEGsystem stream where MPEG video data is multiplexed, that is, a VOB.Therefore, a time map (TMAP) associates the time with the address in aVOB.

In this way, in the case where time information is given, the VOBU towhich the time belongs to is searched first (by following PTSs of VOBUsin order), the PTS immediately before the time is jumped into the VOBUthat a TMAP has (the address specified by I_start), decoding is startedwith the top I picture of the VOBU, and display is started with thepicture corresponding to the time.

Next, the internal structure of a play list information (“XXX.PL”) willbe described with reference to FIG. 26.

The play list information includes a cell list (CellList) and an eventlist (EventList).

The cell list (CellList) is a playback cell sequence in the play list,and cells are played back in the description order indicated in thislist. The contents of the cell list (CellList) is the number of cells(Number) and the information of each cell (Cell#1 to Cell#n).

The cell information (Cell#) has a VOB file name (VOBName), startingtime (In) and end time (Out) in the VOB, and subtitles (SubtitleTable).The starting time (In) and the end time (Out) are represented as a framenumber in each VOB. It is possible to obtain the address of the VOB dataneeded for playback by using the above-described time map (TMAP).

The subtitle table (SubtitleTable) is a table storing subtitleinformation that is played back synchronously with the VOB. Like in thecase of audio, plural languages are included in subtitles. The firstinformation of the subtitle table (SubtitleTable) includes the number oflanguages (Number) and the following tables (Language#1 to Language#k)prepared in a unit of a language.

Each language table (Language#) includes language information (Lang),the number (Number) of items of subtitle information of subtitles to bedisplayed separately, and subtitle information (Speech#1 to Speech#j) ofsubtitles to be displayed separately. The subtitle information (Speech#)includes an image data file name (Name), subtitle display starting time(In), subtitle display ending time (Out) and a subtitle display position(Position).

The event list (EventList) is a table defining each event that occurs inthe play list. The event list includes the number of events (Number) andrespective events (Event#1 to Event#m). Each event (Event#) includes anevent type (Type), an event ID (ID), an event occurrence time (Time) andan event duration (Duration).

FIG. 27 is an event handler table (“XXX.PROG”) having an event handler(that is a time event and a user event for menu selection) prepared on aplay list basis.

The event handler table includes the number of defined eventhandlers/programs (Number) and the respective event handlers/programs(Program#1 to Program#n). The contents of each event hander/program(Program#) is the definition of the start of an event handler(<event_handler> tag) and the event hander ID (ID) that is paired withthe earlier described event ID, and next to it, the program described in“{ }” that follows Function. The event (Event#1 to Event#m) stored inthe event list (EventList) of the earlier-described “XXX.PL” isspecified using an ID (ID) of the event handler of “XXX.PROG”.

Next, the internal structure of the information concerning the whole BDdisc (“BD.INFO”) will be described with reference to FIG. 28.

The information concerning the whole BD disc includes a title list(TitleList) and an event table for global event (EventList).

The title list (TitleList) includes the number of titles of a disc(Number) and items of title information (Title#1 to Title#n) thatfollows the number of titles. The respective items of title information(Title#) include a play list table included in the title (PLTable) and achapter list in the title (ChapterList). The play list table (PLTable)includes the number of play lists in the title (Number) and play listnames (Name) that are the file names of play lists.

The chapter list (ChapterList) includes the number of chapters includedin the title (Number) and items of chapter information (Chapter#1 toChapter#n). Each item of chapter information (Chapter#) includes a celltable (CellTable) included in the chapter, and the cell table(CellTable) includes the number of cells (Number) and items of cellentry information (CellEntry#1 to CellEntry#k). The cell entryinformation (CellEntry#) includes the play list name including the celland a cell number in the play list.

The event list (EventList) includes the number of global events (Number)and items of global event information. It should be noted that theglobal event to be defined first is called first event (FirstEvent), andis the event called first after the BD disc is inserted into a player.The event information for global event has only an event type (Type) andan event ID (ID).

FIG. 29 is a table (“BD.PROG”) of a program of a global event handler.The contents of this table is the same as the contents of the eventhandler table described in FIG. 27.

In the case of multiplexing output data of the multiplexing apparatusesin the first and second embodiments using the BD-ROM format like above,a VOBU is made up of one or more random access unit RAUs, and theplayback order of the clips are specified in a play list. Here, framedisplay information can be shown by BD management information. Forexample, frame delay information may be stored in play items of a playlist, or a table indicating access information such as an EP map. Also,frame delay information may be stored in a table indicating theattribute information of the coded stream. Further, in the case ofindicating (i) the maximum value of frame delays of the coded stream tobe stored in the multiplexed data, (ii) frame delay common among all thecoded streams, or the like, frame delays may be indicated as theinformation higher than the information concerning the respective codedstreams.

Note that the playback order of clips may be determined based on theinformation different from the play list or a predetermined order.

Note that the access information such as EP map may be stored in a tableas binary data, or in the form of text data which may be an XML(Extensible Markup Language) and the like.

Fifth Embodiment

FIG. 30 is a block diagram roughly showing the functional structure of aplayer that plays back a BD disc concerning a fifth embodiment.

The data on the BD disc 201 is read out through an optical pick up 202.The read out data is transmitted to an exclusive memory depending on thetypes of the respective data. The BD playback program (the contents of“BD.PROG” or “XXX.PROG”) is transmitted to a program memory 203. Also,the BD management information (“BD.INFO”, “XXX.PL” or “YYY.VOBI”) istransmitted to a management information memory 204. Also, the AV data(“YYY.VOB” or “ZZZ.PNG”) is transmitted to an AV memory 205.

The BD playback program recorded in the program memory 203 is processedby a program processing unit 206. Also, the BD management informationrecorded in the management information memory 204 is processed by themanagement information processing unit 207. Also, the AV data recordedin the AV memory 205 is processed by a presentation processing unit 208.

The program processing unit 206 receives the information of play liststo be played back by the management information processing unit 207 andthe event information such as the execution timing of the program andperforms the processing of the program. Also, it is possible todynamically change the play lists to be played back by the program. Thiscan be realized by sending a playback instruction of the play lists tothe management information processing unit 207. The program processingunit 206 receives an event from a user, in other words, receives arequest through a remote controller, and in the case where there is aprogram corresponding to the user event, it executes the program.

The management information processing unit 207 receives an instructionfrom the program processing unit 206, analyzes the play lists and themanagement information of the VOBs corresponding to the play lists, andinstructs the presentation processing unit 208 to play back the targetAV data. Also, the management information processing unit 207 receivesthe standard time information from the presentation processing unit 208,instructs the presentation processing unit 208 to stop playing back theAV data based on the time information. Also, the management informationprocessing unit 207 generates an event to notify the program processingunit 206 of the program execution timing.

The presentation processing unit 208 has a decoder that can processvideo, audio, subtitles/images (still pictures) respectively. It decodesand outputs the AV data according to an instruction from the managementinformation processing unit 207. In the case of video data, andsubtitles/images, they are decoded and then rendered in the respectiveexclusive planes, that is, the video plane 210 and the image plane 209.After that, the synthesis processing unit 211 performs the synthesisprocessing on the video, and outputs the video to a display device suchas a TV.

The presentation processing unit 208 interprets the user operationrequesting for multi-angle playback or digest playback at the time ofmulti-angle playback or digest playback, and notifies the managementinformation processing unit 207 of the information concerning an anglechangeable point or the like. The management information processing unit207 determines frame delay at the time of playback based on the framedelay information of the clips to be played back, and notifies thepresentation processing unit of the frame delay.

Note that access information such as an EP map may be stored in a tableas a binary data and may also be in a text format such as an ExtensibleMarkup Language (XML).

Sixth Embodiment

Further, it becomes possible to easily execute the processing shown inthe respective embodiments in an independent computer system, byrecording the program for realizing the multiplexing method and thedemultiplexing method shown in the above-described embodiments into arecording medium such as a flexible disc.

FIG. 31C is an illustration of how the computer system executes themoving picture coding method and the moving picture decoding method ofthis embodiment using a program recorded in a recording medium such as aflexible disc.

FIG. 31A shows an example of a physical format of a flexible disc as arecording medium. FIG. 31B shows a flexible disc and the front view andthe cross-sectional view of the appearance of the flexible disc. Aflexible disc (FD) is contained in a case F, a plurality of tracks (Tr)are formed concentrically on the surface of the disc from the outerradius into the inner radius of the disc, and each track is divided into16 sectors (Se) in the angular direction. Therefore, in the case of theflexible disc storing the above-described program, the program isrecorded in an area allocated for it on the flexible disc (FD).

Also, FIG. 31C shows the structure for recording and playing back theprogram on the flexible disc. In the case of recoding the above programfor realizing the multiplexing method and the demultiplexing method onthe flexible disc FD, a computer system Cs writes the program on theflexible disc through a flexible disc drive. Also, in the case ofconstructing a multiplexing apparatus and a demultiplexing apparatus forrealizing the multiplexing method and the demultiplexing method usingthe program in the flexible disc, the program is read out from theflexible disc through the flexible disc drive, and it is transmitted tothe computer system.

Note that the above description has been made using a flexible disc as arecording medium, but the program can be recorded on an optical disc.Also, a recording medium is not limited to this, other recording mediumsuch as an IC card, a ROM cassette can be used as long as it can recordthe program.

Up to this point, the multiplexing apparatus, the demultiplexingapparatus, the BD disc player and the like concerning the presentinvention have already been described based on the above-describedrespective embodiments, but the present invention is not limited tothese embodiments. The present invention includes variations of theseembodiments that a person skilled in the art would conceive as long asthey are within the scope of the subject matter of the presentinvention.

For example, the following are included in the present invention: (i) anoptical disc recording apparatus including a multiplexing apparatus inthis embodiment; a moving picture sending apparatus; a digitaltelevision broadcasting apparatus; a Web server; a communicationapparatus: a mobile information terminal and the like; and (ii) a movingpicture receiving apparatus including a demultiplexing apparatus in thisembodiment; a digital television broadcasting receiving apparatus; acommunication apparatus; a mobile information terminal and the like.

Note that the respective functional blocks of the block diagrams (FIG.14, FIG. 17, FIG. 21 and the like) are typically realized as a LargeScale Integration (LSI) that is an integrated circuit. Each of them maybe made into a chip. Also, one, some or all of them may be integratedinto a chip (For example, functional blocks other than a memory may beintegrated into a chip).

Here, the circuit is called LSI, but it may be called Integrated Circuit(IC), system LSI, super LSI or ultra LSI depending on the degree ofintegration.

Also, making method of an integration circuit is not limited to this, anexclusive circuit or a general processor may be realized. Also, it ispossible to use a Field Programmable Gate Array (FPGA) that isprogrammable after making an LSI or a reconfigurable processor that iscapable of reconfiguring the connection or settings of the circuit cellinside the LSI.

Further, when a making method of an integrated circuit that replaces anLSI is conceived resulting from the development of semiconductortechnique or the emergence of a derivative technique, functional blocksmay be integrated using such a technique as a matter of course.Adaptation of bio technique or the like is likely.

Also, among the respective functional blocks, only the mean for storingdata to be coded or recoded may be configured separately without beingintegrated into a chip.

Although only an exemplary embodiment of this invention has beendescribed in detail above, those skilled in the art will readilyappreciate that many modifications are possible in the exemplaryembodiment without materially departing from the novel teachings andadvantages of this invention. Accordingly, all such modifications areintended to be included within the scope of this invention.

INDUSTRIAL APPLICABILITY

The multiplexing method and the demultiplexing method of the presentinvention can be applied for general apparatuses that have a specialplayback function such as multi-angle playback and digest playback, andthus they are especially effective in the playback of package media inwhich an MPEG-4 AVC coded stream is multiplexed.

1-16. (canceled)
 17. A multiplexing apparatus for multiplexing codedstreams with other information, the coded streams including codedpictures on a per randomly-accessible access unit basis, saidmultiplexing apparatus comprising: a first determining unit configuredto determine delay amounts in two access units to be decodedsequentially, which are a first access unit and a second access unit, inthe coded streams such that the delay amounts in the two access unitsbecome equal to each other in the case where the two access units areaccess units each having a different angle and are decoded sequentiallychanging angles from an angle of the first access unit which is aswitching source to an angle of the second access unit which is aswitching destination or in the case where the two access units areaccess units each having a same angle and are decoded sequentially, thedelay amounts each being a time lag from a decoding time of a toppicture in decoding order to a display time of a top picture in displayorder, the second access unit being selected from among access unitswhich have identical playback start times and identical playback endtimes in the coded streams; a generating unit configured to code thepictures to be included in the two access units and generate the codedstreams according to the delay amounts determined by said firstdetermining unit; and a multiplexing unit configured to multiplex thecoded streams generated by said generating unit and the otherinformation, wherein the other information includes a flag indicatingwhether or not the two access units can be connected to each otherseamlessly, and the flag is determined depending on whether or not thedelay amounts in the two access units are equal to each other.
 18. Aplayback apparatus which demultiplexes the multiplexed data generated bythe multiplexing apparatus according to claim 17 and displays thedemultiplexed data, said playback apparatus comprising: a flagdemultiplexing unit configured to demultiplex, from the multiplexeddata, the flag in the case where the two access units are the accessunits each having the different angle and are decoded sequentiallychanging angles from the angle of the first access unit which is theswitching source to the angle of the second access unit which is theswitching destination or in the case where the two access units are theaccess units each having the same angle and are decoded sequentially,the second access unit being selected from among the access units whichhave the identical playback start times and the identical playback endtimes in the coded streams; a second determining unit configured todetermine the delay amount in a top access unit at a display startingtime in the case where the flag demultiplexed by said flagdemultiplexing unit indicates that the two access units can be connectedto each other seamlessly; and a display unit configured to sequentiallydecode the two access units each having the different angle changingangles from the angle of the first access unit to the angle of thesecond access unit or sequentially decode the two access units eachhaving the same angle, according to the delay amount determined by saidsecond determining unit, and display the decoded two access units.
 19. Amultiplexing method for multiplexing coded streams with otherinformation, the coded streams including coded pictures on a perrandomly-accessible access unit basis, said multiplexing methodcomprising: determining delay amounts in two access units to be decodedsequentially, which are a first access unit and a second access unit, inthe coded streams such that the delay amounts in the two access unitsbecome equal to each other in the case where the two access units areaccess units each having a different angle and are decoded sequentiallychanging angles from an angle of the first access unit which is aswitching source to an angle of the second access unit which is aswitching destination or in the case where the two access units areaccess units each having a same angle and are decoded sequentially, thedelay amounts each being a time lag from a decoding time of a toppicture in decoding order to a display time of a top picture in displayorder, the second access unit being selected from among access unitswhich have identical playback start times and identical playback endtimes in the coded streams; coding the pictures to be included in thetwo access units and generating the coded streams according to the delayamounts determined in said determining; and multiplexing the codedstreams generated in said generating and the other information, whereinthe other information includes a flag indicating whether or not the twoaccess units can be connected to each other seamlessly, and the flag isdetermined depending on whether or not the delay amounts in the twoaccess units are equal to each other.
 20. A playback method fordemultiplexing the multiplexed data generated using the multiplexingmethod according to claim 19, said playback method comprising:demultiplexing the flag from the multiplexed data in the case where thetwo access units are the access units each having the different angleand are decoded sequentially changing angles from the angle of the firstaccess unit which is the switching source to the angle of the secondaccess unit which is the switching destination or in the case where thetwo access units are the access units each having the same angle and aredecoded sequentially, the second access unit being selected from amongthe access units which have the identical playback start times and theidentical playback end times in the coded streams, the multiplexed dataincluding (i) coded streams including coded pictures on a perrandomly-accessible access unit basis and (ii) other informationincluding the flag; determining a delay amount in a top access unit at adisplay starting time in the case where the flag demultiplexed in saiddemultiplexing indicates that the two access units can be connected toeach other seamlessly; and sequentially decoding the two access unitseach having the different angle changing angles from the angle of thefirst access unit to the angle of the second access unit or sequentiallydecoding the two access units each having the same angle, according tothe delay amount determined in said determining, and displaying thedecoded two access units.
 21. A recording method for recording, onto arecording medium, multiplexed data including coded streams and otherinformation, the coded streams including coded pictures on a perrandomly-accessible access unit basis, said recording method comprising:determining delay amounts in two access units to be decodedsequentially, which are a first access unit and a second access unit, inthe coded streams such that the delay amounts in the two access unitsbecome equal to each other in the case where the two access units areaccess units each having a different angle and are decoded sequentiallychanging angles from an angle of the first access unit which is aswitching source to an angle of the second access unit which is aswitching destination or in the case where the two access units areaccess units each having a same angle and are decoded sequentially, thedelay amounts each being a time lag from a decoding time of a toppicture in decoding order to a display time of a top picture in displayorder, the second access unit being selected from among access unitswhich have identical playback start times and identical playback endtimes in the coded streams; coding the pictures to be included in thetwo access units and generating the coded streams according to the delayamounts determined in said determining; multiplexing the coded streamsgenerated in said generating and the other information; and recordingthe multiplexed data multiplexed in said multiplexing onto a recordingmedium, wherein the other information includes a flag indicating whetheror not the two access units can be connected to each other seamlessly,and the flag is determined depending on whether or not the delay amountsin the two access units are equal to each other.